feat(rocm): MoE combine/gate ops — SharedExpertGate, MoeCombine, MoeCombineGate (issue #41) - #509

…ms -- the mudler#509 review rework

Review sweep findings (localai-bot, 2026-08-13), all accepted:

1. The donor's dtype refusals were dropped: cuda_moe.cu:520-524/:597-604 open
   with VT_CHECKs refusing non-f32/bf16; without them an f16 expert_out passes
   the seam's IsFloat gate and Tensor::Ptr<T>()'s unchecked cast reads 4 bytes
   per element from a 2-byte allocation. Refusals added to all three ROCm
   entry points (SharedExpertGate included: its 4-arm dispatch has the same
   f16 hazard on sd).
2. The case now exercises the PRODUCTION dtype mix, not only f32: the model
   path runs expert_out bf16 (qwen3_5.cpp DBuf ddown), shared bf16, out bf16.
   The new bf16 arm is asserted BIT-EXACT against the CPU reference (both
   sides thread-per-element, same sequential K order, single store rounding,
   -ffp-contract=off), and the f32 MoeCombine arm is tightened from NMSE to
   bit-exact per the donor's design comment (cuda_moe.cu:465-468). Writing the
   bf16 arm caught a construction bug in the first version of it (an f32-typed
   tensor over a bf16 buffer -- exactly the OOB class the review predicted the
   missing arm hid); fixed and re-verified against an independent host
   composite (0/320) plus a raw-hipMalloc scratch replica of both backends.
3. CMakeLists.txt: the mangled duplicate rocm_moe_chain.hip line removed.
4. docs/FEATURES.md op count 44 -> 47 (counted: the registration sites).

Gates (gfx1100, flock): test_backend_cross_device 20/20 (bit-exact bf16 +
f32 arms).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: pi:kimi-k3 [pi]

…ms -- the mudler#509 review rework

Review sweep findings (localai-bot, 2026-08-13), all accepted:

1. The donor's dtype refusals were dropped: cuda_moe.cu:520-524/:597-604 open
   with VT_CHECKs refusing non-f32/bf16; without them an f16 expert_out passes
   the seam's IsFloat gate and Tensor::Ptr<T>()'s unchecked cast reads 4 bytes
   per element from a 2-byte allocation. Refusals added to all three ROCm
   entry points (SharedExpertGate included: its 4-arm dispatch has the same
   f16 hazard on sd).
2. The case now exercises the PRODUCTION dtype mix, not only f32: the model
   path runs expert_out bf16 (qwen3_5.cpp DBuf ddown), shared bf16, out bf16.
   The new bf16 arm is asserted BIT-EXACT against the CPU reference (both
   sides thread-per-element, same sequential K order, single store rounding,
   -ffp-contract=off), and the f32 MoeCombine arm is tightened from NMSE to
   bit-exact per the donor's design comment (cuda_moe.cu:465-468). Writing the
   bf16 arm caught a construction bug in the first version of it (an f32-typed
   tensor over a bf16 buffer -- exactly the OOB class the review predicted the
   missing arm hid); fixed and re-verified against an independent host
   composite (0/320) plus a raw-hipMalloc scratch replica of both backends.
3. CMakeLists.txt: the mangled duplicate rocm_moe_chain.hip line removed.
4. docs/FEATURES.md op count 44 -> 47 (counted: the registration sites).

Gates (gfx1100, flock): test_backend_cross_device 20/20 (bit-exact bf16 +
f32 arms).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: pi:kimi-k3 [pi]

…ms -- the mudler#509 review rework

Review sweep findings (localai-bot, 2026-08-13), all accepted:

1. The donor's dtype refusals were dropped: cuda_moe.cu:520-524/:597-604 open
   with VT_CHECKs refusing non-f32/bf16; without them an f16 expert_out passes
   the seam's IsFloat gate and Tensor::Ptr<T>()'s unchecked cast reads 4 bytes
   per element from a 2-byte allocation. Refusals added to all three ROCm
   entry points (SharedExpertGate included: its 4-arm dispatch has the same
   f16 hazard on sd).
2. The case now exercises the PRODUCTION dtype mix, not only f32: the model
   path runs expert_out bf16 (qwen3_5.cpp DBuf ddown), shared bf16, out bf16.
   The new bf16 arm is asserted BIT-EXACT against the CPU reference (both
   sides thread-per-element, same sequential K order, single store rounding,
   -ffp-contract=off), and the f32 MoeCombine arm is tightened from NMSE to
   bit-exact per the donor's design comment (cuda_moe.cu:465-468). Writing the
   bf16 arm caught a construction bug in the first version of it (an f32-typed
   tensor over a bf16 buffer -- exactly the OOB class the review predicted the
   missing arm hid); fixed and re-verified against an independent host
   composite (0/320) plus a raw-hipMalloc scratch replica of both backends.
3. CMakeLists.txt: the mangled duplicate rocm_moe_chain.hip line removed.
4. docs/FEATURES.md op count 44 -> 47 (counted: the registration sites).

Gates (gfx1100, flock): test_backend_cross_device 20/20 (bit-exact bf16 +
f32 arms).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: pi:kimi-k3 [pi]

…ms -- the mudler#509 review rework

Review sweep findings (localai-bot, 2026-08-13), all accepted:

1. The donor's dtype refusals were dropped: cuda_moe.cu:520-524/:597-604 open
   with VT_CHECKs refusing non-f32/bf16; without them an f16 expert_out passes
   the seam's IsFloat gate and Tensor::Ptr<T>()'s unchecked cast reads 4 bytes
   per element from a 2-byte allocation. Refusals added to all three ROCm
   entry points (SharedExpertGate included: its 4-arm dispatch has the same
   f16 hazard on sd).
2. The case now exercises the PRODUCTION dtype mix, not only f32: the model
   path runs expert_out bf16 (qwen3_5.cpp DBuf ddown), shared bf16, out bf16.
   The new bf16 arm is asserted BIT-EXACT against the CPU reference (both
   sides thread-per-element, same sequential K order, single store rounding,
   -ffp-contract=off), and the f32 MoeCombine arm is tightened from NMSE to
   bit-exact per the donor's design comment (cuda_moe.cu:465-468). Writing the
   bf16 arm caught a construction bug in the first version of it (an f32-typed
   tensor over a bf16 buffer -- exactly the OOB class the review predicted the
   missing arm hid); fixed and re-verified against an independent host
   composite (0/320) plus a raw-hipMalloc scratch replica of both backends.
3. CMakeLists.txt: the mangled duplicate rocm_moe_chain.hip line removed.
4. docs/FEATURES.md op count 44 -> 47 (counted: the registration sites).

Gates (gfx1100, flock): test_backend_cross_device 20/20 (bit-exact bf16 +
f32 arms).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: pi:kimi-k3 [pi]

Fold research 6195: ProductGetBlasStreamIsCapturing calls the exact
HipBlasHooks hook. HIP product probe begins capture and asserts true;
always-false hook mutation is RED. Host fake-capture case unchanged.

1a1153d6 is not a review target. Adjacent mudler#785/mudler#523/mudler#509/mudler#834 noted
in spec; no pickup.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Hermes:grok-4.6 [Hermes]

Fold research 6195: ProductGetBlasStreamIsCapturing calls the exact
HipBlasHooks hook. HIP product probe begins capture and asserts true;
always-false hook mutation is RED. Host fake-capture case unchanged.

1a1153d6 is not a review target. Adjacent mudler#785/mudler#523/mudler#509/mudler#834 noted
in spec; no pickup.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Hermes:grok-4.6 [Hermes]

… OFF

Implements [`.agents/specs/rocm-decode-attn-d128.md`](https://github.com/mudler/vllm.cpp/blob/main/.agents/specs/rocm-decode-attn-d128.md), which landed ahead of this change as #564.

The ROCm half of #382. The CUDA half merged as #425 (`66399617`); this mirrors it, adopting that arm's flag, default and stated reason rather than inventing new ones.

## What changes

`d == 128` — the Qwen3-dense / Llama / Mistral head size — reaches the fast decode kernels instead of falling through to the generic `PagedAttnOnline`. `LoadRowEplBf16`/`StoreRowEplBf16` gain an `EPL=4` (`uint2`) case beside the existing `EPL=8`/`16`; the dispatch gates and the two launch switches gain a `d == 128` arm. No new kernel and no new algorithm — the kernel bodies were already generic over `EPL`.

**Default OFF, opt in with `VT_ATTN_DECODE_D128=1`** — the same env var, default and reason as the merged CUDA arm. The arm is correctness-complete but not byte-exact against the kernel it replaces: warp-strided online softmax reduces the KV sequence in a different **order** than `PagedAttnOnline`'s per-tile loop, so a greedy anchor can move at an exact bf16 tie. Shipping OFF keeps every existing golden byte-identical. The flip owes the near-tie razor, a distributional gate and regen under the ratified-tie rule, and per the spec must be argued **per backend** — see the reversal below. That is what keeps #382 open.

## Reviewer note

Spec §4 item 3 writes the gate as `(d == 128 && (decode_d128 || decode_wmma))`. This commit implements it **without** the `decode_wmma` disjunct, which is what the same item's "Forward reference" paragraph instructs: `VT_ATTN_DECODE_WMMA` does not exist in the tree, and the flag lands with the rocWMMA arm on its own branch. The difference is intentional; it is visible in the diff before the note explaining it is.

## Evidence

gfx1200 (RX 9060 XT, RDNA4, 32 CU), ROCm 7.2.3, `$GPU_LOCK` held. All figures are a same-binary flag A/B — no rebuild between arms — at 1024-token synthetic prompt, 128 generated, greedy, seed 0, 2 reps per cell agreeing within ~1%.

| Model | head_dim | decode path | TPOT OFF | TPOT ON | speedup |
|---|---|---|---|---|---|
| Qwen3-0.6B | 128 | `qg=2` fused | 42.53 ms | 11.78 ms | **3.61x** |
| Qwen3-1.7B | 128 | `qg=2` fused | 52.85 ms | 21.93 ms | **2.41x** |
| Qwen3-4B | 128 | `qg=4` per-head | 81.89 ms | 39.22 ms | **2.09x** |
| Qwen3.5-0.8B | 256 | — (control) | 23.76 ms | 23.55 ms | 1.01x |

Qwen3-4B has no GQA fusion at any head_dim, so its 2.09x isolates the `EPL` widening from the fusion.

**Qwen3.5-0.8B is the negative control and it earned its keep.** Its `head_dim` is 256, so the `d == 128` gate provably cannot reach it. Its first OFF rep came in a 33% outlier at 31.14 ms, which a blind 2-rep average would have reported as a ~1.2x "win" for a model the flag cannot affect. Re-run three times: 23.86 / 23.75 / 23.68 against ON's 23.52 / 23.57.

End-to-end output throughput rises less than TPOT on the same runs (0.6B 2.48x, 1.7B 2.05x, 4B 2.02x) because they carry a 1024-token prefill the flag does not touch. TPOT isolates decode; throughput dilutes it.

### Concurrency — the advantage grows, it does not compress

Qwen3-1.7B, `--num-prompts` = 2x concurrency:

| Conc | tok/s OFF | tok/s ON | ratio | TPOT ratio |
|---|---|---|---|---|
| 1 | 12.89 | 24.66 | 1.91x | 2.40x |
| 2 | 23.27 | 47.45 | 2.04x | 2.45x |
| 4 | 39.10 | 86.86 | 2.22x | 2.46x |
| 8 | 58.97 | 147.35 | **2.50x** | **2.77x** |
| 16 | 78.43 | 227.08 | **2.90x** | **3.18x** |

This refuted the prediction made before the run, which reasoned that a tiny grid at concurrency 1 flatters the fast kernel. The dominant effect is the reverse: from c8 to c16 the fallback scales only **1.33x** against the arm's **1.54x**, and scaling efficiency at c16 relative to perfect-linear-from-c1 is **38% OFF against 58% ON**. `PagedAttnOnline` is the batch-scaling bottleneck, not merely slow per call, so the win is largest in the regime a server actually runs in.

The c1 row reproduces an independently-run four-model sweep to within ~1% (52.85/21.93 there vs 53.40/22.26 here).

### Correctness

- `ctest -R 'rocm|cross_device'` **5/5**, chained directly to the exact-SHA push.
- New case: "paged attention at Qwen3 geometry (bf16, GQA 2, head_dim 128) matches the CPU oracle", looped over `RegisteredDevices()`, NMSE <= 5e-4 vs the CPU oracle plus `OpProviderStats::declines == 0`. Genuinely new coverage — the existing generic cross-device test runs at `d=8, f32` and never reached any bf16 `EPL`-templated kernel, so none of them had bf16 coverage in this suite. (The merged CUDA arm shipped with no test at all.)
- Because the arm ships OFF **and** its flag is read into a `static const bool` — once per process — the default registration can only ever gate the fallback. `tests/CMakeLists.txt` adds a second invocation with the flag set, same shape as the existing `test_dense_gateup_fused_marlin_off_*` pair. Verified non-vacuous against the #463 trap: 1 case, 6 assertions, not zero.
- Full `ctest` 448/455. The 7 failures are **proven** pre-existing, not asserted: a clean `main` `2784dd7b` worktree built from source with none of this code fails the identical set (only `test_op_parity`'s index shifts 403 -> 404, from the added registration). They are a missing `shellcheck`, an mmap-RSS assertion, a JSON type error, and the `SharedExpertGate` ROCm registration gap owed to unmerged #509.
- `agent-preflight` fails 9, a strict **subset** of that same baseline's 10 (differing only by `role-undeclared`). `check-commit-trailers` and `check-doc-checkpoint` both pass against this base.

## Carried finding

#382 measured this same `EPL=4` arm **1.6x slower** on sm_110 / Jetson AGX Thor, where gfx1200 measures it 2-3.6x faster. Recorded, not reconciled — different kernels, different fallbacks, different memory systems. It is why the default-ON flip must be argued per backend rather than once, and it is preserved in the spec rather than averaged away.

## Against the pinned oracle: 6.35x to 1.75x slower on per-token decode

Measured after the tables above, with **both sides in the same container**, oracle = vLLM `555967922` in its production configuration via `vllm bench serve`. Qwen3-0.6B, 1024 in / 128 out, concurrency 1, **8 prompts**, warmup discarded, **3 reps**:

| | TPOT reps | mean | vs oracle |
|---|---|---|---|
| ours, flag unset | 42.54 / 42.46 / 42.19 | 42.40 ms | 6.35x slower |
| ours, `VT_ATTN_DECODE_D128=1` | 11.97 / 11.38 / 11.66 | **11.67 ms** | **1.75x slower** |
| vLLM `555967922` | 6.57 / 6.90 / 6.58 | 6.68 ms | — |

Running our binary against the container's ROCm rather than the host's is a substitution, so it was proved inert first: in-container matches native at 42.79 vs 42.53 ms unset, and 12.03 vs 11.78 ms with the flag.

The prompt count is load-bearing. At `--num-prompts 2` the oracle returned TPOT **6.96 ms and 13.45 ms on consecutive reps**, a ~2x spread averaging to a plausible-looking and entirely fictional number. At 8 prompts with a discarded warmup both sides hold to ~±0.3 ms.

This number lived only in a PR comment, which a squash merge does not carry into the tree. It is now in the spec's §5 and appended to `.agents/benchmark-record.md`, with its caveats attached rather than trailing.

## The container/glibc blocker was RETRACTED

An earlier revision of this body, and the spec's §6, said the oracle re-measure was blocked on a Nix-glibc vs container-glibc ABI mismatch. **That diagnosis was wrong and is retracted.** Our binary runs inside the pinned oracle container; the earlier failures were self-inflicted (`LD_LIBRARY_PATH` exported container-wide, which breaks the container's own tools, plus a bind mount that silently yielded nothing and looked exactly like a missing ELF interpreter). §6 now reads "not run — **not blocked**", and the WMMA-spec cross-reference is gone. A false blocker in the record is worse than no record, because it stops the next person from trying.

## Not claimed

**This does not close #488.** That issue asks for a **per-call** kernel comparison and explicitly asserts no cause. The number above is **per-token latency** with asymmetric harnesses — the oracle over HTTP via `vllm bench serve`, ours in-process — so TPOT is the only comparable axis, and TTFT, E2EL and end-to-end throughput carry the oracle's HTTP and tokenizer overhead and are directional only. It is not the same-tool per-call trace `AGENTS.md` wants before a throughput claim. `rocprofv3` is present in the container and our binary traces under it; what is still owed is decode-phase windowing on the oracle side, or the trace compares our decode against vLLM's model load and graph capture. One board, one shape. **`docs/BENCHMARKS.md`'s ROCm axis stays PENDING**, and the row this PR adds is marked DIRECTIONAL and sits beside the existing row rather than overwriting it.

**The flag-ON arm still has no proof it REACHES the new kernel, now filed as #1134.** `RegisteredDevices()` (`tests/vt/test_backend_cross_device.cpp:84-96`) enumerates `{kCUDA, kMETAL, kVULKAN, kXPU, kROCM}` and excludes `kCPU`, so on a CPU-only runner — which is what CI has — the new case reports 1 test case, 0 assertions, exit 0, for **both** registrations. On ROCm the case's only backend assertion is `OpProviderStats::declines == 0`, counted at **provider** granularity, so it is identical with the flag set and unset. §9 stop condition 2 is left OPEN. The spec disclosed this honestly; what was missing is the issue `AGENTS.md` requires for a known gap not fixed in flow. Searched before filing: not a duplicate of #463 (the unset-weights-env-var shape, which does not describe the `declines` half), #785 (a kernel that never LAUNCHES behind a dead `#if`) or #900 (same family, LTX-2.5 subject).

Also out of scope and named in the spec's new `## Owed` section: the dtype gap (ROCm's decode-opt is bf16-only at every head_dim, so 4 of 5 dtype combinations still fall to `PagedAttnOnline` at `d=128` — pre-existing, inherited, not introduced), `qg=4`/`qg=8` fusion, `d=128` prefill, and the rocWMMA arm.

## Record repairs carried in the final commit

`docs(BACKEND-ROCM): retract the blocker, keep the oracle number, and file the gap`, on top of joral's commits, which are untouched. It carries the retraction above; the oracle number into §5 and `.agents/benchmark-record.md`; `docs/BENCHMARKS.md` and `docs/STATUS.md` reconciled; §7's stale "**two** flag-on ctest registrations" corrected to one, matching what `8aedd780` already fixed in §4 item 3 and the Test-coverage section; a literal `## Owed` heading over the owed list; and #1134 filed and appended to `.agents/issue-index.md`. Two comment-only edits at `rocm_paged_attn.hip:330` and `:455`, which still enumerated "EPL=8 → d=256, EPL=16 → d=512" without the new `EPL=4` case although the top-of-file comment at `:264` had been updated.

The branch is **rebased** onto `origin/main` `d1e5e9bc` — it was 39 commits behind, and the rebase drops the earlier `merge: upstream/main` commit. Gates rerun from the worktree with explicit SHAs: `check-commit-trailers`, `check-commit-style`, `check-doc-checkpoint`, `check-public-doc-tables`, `check-agent-record` and `check-pr-size`, all OK. Not rebuilt and not re-run on hardware: every gfx1200 figure here is joral's, unchanged.

Row: BACKEND-ROCM
Issue: #382
Issue: #1134
Spec: #564

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]

…ms -- the mudler#509 review rework

Review sweep findings (localai-bot, 2026-08-13), all accepted:

1. The donor's dtype refusals were dropped: cuda_moe.cu:520-524/:597-604 open
   with VT_CHECKs refusing non-f32/bf16; without them an f16 expert_out passes
   the seam's IsFloat gate and Tensor::Ptr<T>()'s unchecked cast reads 4 bytes
   per element from a 2-byte allocation. Refusals added to all three ROCm
   entry points (SharedExpertGate included: its 4-arm dispatch has the same
   f16 hazard on sd).
2. The case now exercises the PRODUCTION dtype mix, not only f32: the model
   path runs expert_out bf16 (qwen3_5.cpp DBuf ddown), shared bf16, out bf16.
   The new bf16 arm is asserted BIT-EXACT against the CPU reference (both
   sides thread-per-element, same sequential K order, single store rounding,
   -ffp-contract=off), and the f32 MoeCombine arm is tightened from NMSE to
   bit-exact per the donor's design comment (cuda_moe.cu:465-468). Writing the
   bf16 arm caught a construction bug in the first version of it (an f32-typed
   tensor over a bf16 buffer -- exactly the OOB class the review predicted the
   missing arm hid); fixed and re-verified against an independent host
   composite (0/320) plus a raw-hipMalloc scratch replica of both backends.
3. CMakeLists.txt: the mangled duplicate rocm_moe_chain.hip line removed.
4. docs/FEATURES.md op count 44 -> 47 (counted: the registration sites).

Gates (gfx1100, flock): test_backend_cross_device 20/20 (bit-exact bf16 +
f32 arms).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: pi:kimi-k3 [pi]

…ms -- the mudler#509 review rework

Review sweep findings (localai-bot, 2026-08-13), all accepted:

1. The donor's dtype refusals were dropped: cuda_moe.cu:520-524/:597-604 open
   with VT_CHECKs refusing non-f32/bf16; without them an f16 expert_out passes
   the seam's IsFloat gate and Tensor::Ptr<T>()'s unchecked cast reads 4 bytes
   per element from a 2-byte allocation. Refusals added to all three ROCm
   entry points (SharedExpertGate included: its 4-arm dispatch has the same
   f16 hazard on sd).
2. The case now exercises the PRODUCTION dtype mix, not only f32: the model
   path runs expert_out bf16 (qwen3_5.cpp DBuf ddown), shared bf16, out bf16.
   The new bf16 arm is asserted BIT-EXACT against the CPU reference (both
   sides thread-per-element, same sequential K order, single store rounding,
   -ffp-contract=off), and the f32 MoeCombine arm is tightened from NMSE to
   bit-exact per the donor's design comment (cuda_moe.cu:465-468). Writing the
   bf16 arm caught a construction bug in the first version of it (an f32-typed
   tensor over a bf16 buffer -- exactly the OOB class the review predicted the
   missing arm hid); fixed and re-verified against an independent host
   composite (0/320) plus a raw-hipMalloc scratch replica of both backends.
3. CMakeLists.txt: the mangled duplicate rocm_moe_chain.hip line removed.
4. docs/FEATURES.md op count 44 -> 47 (counted: the registration sites).

Gates (gfx1100, flock): test_backend_cross_device 20/20 (bit-exact bf16 +
f32 arms).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: pi:kimi-k3 [pi]

…ms -- the mudler#509 review rework

Review sweep findings (localai-bot, 2026-08-13), all accepted:

1. The donor's dtype refusals were dropped: cuda_moe.cu:520-524/:597-604 open
   with VT_CHECKs refusing non-f32/bf16; without them an f16 expert_out passes
   the seam's IsFloat gate and Tensor::Ptr<T>()'s unchecked cast reads 4 bytes
   per element from a 2-byte allocation. Refusals added to all three ROCm
   entry points (SharedExpertGate included: its 4-arm dispatch has the same
   f16 hazard on sd).
2. The case now exercises the PRODUCTION dtype mix, not only f32: the model
   path runs expert_out bf16 (qwen3_5.cpp DBuf ddown), shared bf16, out bf16.
   The new bf16 arm is asserted BIT-EXACT against the CPU reference (both
   sides thread-per-element, same sequential K order, single store rounding,
   -ffp-contract=off), and the f32 MoeCombine arm is tightened from NMSE to
   bit-exact per the donor's design comment (cuda_moe.cu:465-468). Writing the
   bf16 arm caught a construction bug in the first version of it (an f32-typed
   tensor over a bf16 buffer -- exactly the OOB class the review predicted the
   missing arm hid); fixed and re-verified against an independent host
   composite (0/320) plus a raw-hipMalloc scratch replica of both backends.
3. CMakeLists.txt: the mangled duplicate rocm_moe_chain.hip line removed.
4. docs/FEATURES.md op count 44 -> 47 (counted: the registration sites).

Gates (gfx1100, flock): test_backend_cross_device 20/20 (bit-exact bf16 +
f32 arms).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: pi:kimi-k3 [pi]

mudler#509 landed as a squash while this branch carried its commits, so the shared
content no longer had an ancestor in common with main and five paths conflicted.
Every one is the stack breaking, not a disagreement.

CMakeLists.txt and src/vt/rocm/rocm_ops.hip: both hunks are pure additions
against an empty main side, so this branch's lines are kept and mudler#509's, now
arriving from main, are untouched. RegisterOp count measured rather than
assumed: 47 on main, 49 here, the two being kMatmulBTQuant and
kMatmulBTQuantGrouped.

tests/vt/test_backend_cross_device.cpp: main's file already carries the cases
from mudler#506 and mudler#509, so it was taken whole and this branch's own 170-line
addition re-applied on top. All four case families are present and the
translation unit compiles under the project's -Werror flags.

docs/FEATURES.md is a keyed record: main's two ROCm rows were taken whole and
only the op count re-applied, preserving main's "ROCm guide" link text. The
phrasing is shortened because check-public-doc-tables caps a cell at 220
characters and the natural wording reached 238.

docs/USAGE.md was restructured on main into an index while this branch held the
old ~290 KB document; main's side is taken and one troubleshooting entry states
what keep-quant changes for a user.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]

…he MoE-path terminus (#41) (#523)

## Row

`BACKEND-ROCM` — the last kernel blocker for MoE-bearing models on discrete ROCm. Issue #41. **Stacked on #509** (MoE combine/gate chain).

## What changed

NEW `src/vt/rocm/rocm_grouped_gemm.hip` — ports the keep-quant expert GEMM family from `cuda_quant_dot.cu` 1:1:
- `kMatmulBTQuant` (op 74, non-grouped) + `kMatmulBTQuantGrouped` (op 75, grouped over `expert_ids`)
- Q8_0 / Q4_K / Q5_K / Q6_K weight formats (the ones the target GDN-MoE GGUFs carry); Q8_0 + Q8_K activation quantizers
- `DotQ8_0`/`DotQ4K`/`DotQ5K`/`DotQ6K` superblocks, `__dp4a` → portable `Dp4a` (bit-identical integer core), `__shfl_down_sync` reduction
- Registering the non-grouped op is what flips `GgufQuantComputeAvailable()` on ROCm — the grouped op alone leaves the loader dequantizing experts to bf16.

Cross-device case: all four formats vs the CPU keep-quant oracle, valid random blocks + real activations.

## Evidence (4× gfx1100, ROCm 7.14, Release)

- grouped-quant case: **16/16 assertions** across Q8_0/Q4_K/Q5_K/Q6_K, NMSE ≤ 5e-4 vs the CPU keep-quant reference
- `ctest -R 'rocm|cross_device'`: 4/4; full ctest: only the 5 pre-existing host/lane failures (zero new)
- **E2E: Qwen3.6-35B-A3B Q4_K_M (21GB GDN-MoE GGUF) runs end to end on one gfx1100** with keep-quant active — op 74 AND op 75 resolve `vt-native`, **zero CPU-ref fallback**, correct output. Requires `--max-num-seqs 1` to fit one 24GB card (the GDN state pool otherwise pushes past — a residency note, not a kernel defect; confirmed via a backend-Alloc instrumentation run showing genuine cumulative ~23.8 GiB).
- preflight --staged + trailers green

## Speed claims

- [x] This PR makes NO speed claim.

## Honest gaps

- The K-quant formats Q2_K/Q3_K and the IQ2/IQ3 family are NOT ported (throw loudly); the target models don't use them. They follow the same skeleton.
- Activation scratch is a capture-safe, per-stream grow-only store. It never reallocates during reuse; a larger unseen shape grows the owning stream slot before capture.
- The 35B fits only with `--max-num-seqs 1` on 24GB; multi-GPU expert sharding or host streaming is the follow-on for bigger MoE / longer context.

## Current-main integration cutoff (2026-08-18)

Rebased from `f53b0598b2e773308a8870b322e559f5f3d13dab` to `36b5c4c11e3125a0da42800558f8c3ded3648c64` on recorded current-main cutoff `65d6cdaed3e20e9bc70b4f9374fccafefefa7bd0`. Exact commit replay on final PR #509 parent `fa2e6280`; main’s GGUF loader/test changes remain intact. Fresh cutoff-head builds, focused gates, full repository preflight, and independent mutation review all pass. The 35B production smoke exits 0 through the new MoE kernels; it is reachability evidence only, not a token-parity claim. Cutoff main itself declines at the absent `SharedExpertGate`.

## Current-main merge integration (2026-08-18)

Integrated validated main `27d5432f9ef4c68304ea919259e9edc2076e2509` as the second parent. Candidate head: `789c12232aca556699383ae914230198dedef09b`. Production sources merged without conflict. After validation, `origin/main` advanced to `5870cb2bfeee88a689ea8deaaa96884f7aacc091`; `git merge-tree` reports a clean merge, so no further history rewrite is warranted. The additive test resolution retains the MoE, grouped/non-grouped quant, and main #497 composition cases; a fresh repair implementer restored two branch-side closures caught by the first rebuild. M3, M4, and the 47-op MoE projection remain intact.

Fresh independent review: **PASS**. The always-keep-F16 mutation failed 7/6,099 assertions semantically; restored GREEN was 40/40 and 6,099/6,099. Operator gates: cross-device 24/24 (399), GGUF policy 40/40 (6,099). The 35B Q4_K_M production smoke exits 0 with nonempty generation and no provider decline; this is reachability evidence only, not token parity. Full preflight: **PASS**.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: pi:kimi-k3 [pi]
Assisted-by: codex:gpt-5.6-sol [codex]
Assisted-by: pi:gpt-5.6-sol [pi]